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ABSTRACT 

In  this  study,  we  simulated  a  generic  mounted 
crewstation  environment  and  conducted  an  experiment  to 
examine  the  workload  and  performance  of  the  combined 
position  of  gunner  and  robotic  operator.  Results  showed 
that  gunner’s  target  detection  performance  degraded 
significantly  when  s/he  had  to  concurrently  monitor, 
manage,  or  teleoperate  an  unmanned  ground  vehicle 
compared  to  the  baseline  condition  (gunnery  task  only). 
Additionally,  those  with  higher  spatial  ability  (as 
measured  by  Spatial  Orientation  Test)  performed 
significantly  better  than  those  with  lower  spatial  ability. 
For  the  robotic  tasks,  participants  detected  significantly 
fewer  targets  when  their  robotic  asset  was  semi- 
autonomous  instead  of  teleoperated,  indicating  over¬ 
reliance  on  the  aided  target  recognition  capabilities 
available  when  task  load  was  heavy  (i.e.,  concurrent 
performance  of  the  gunnery  task).  Participants’  perceived 
workload  increased  consistently  as  the  concurrent  task 
conditions  became  more  challenging.  Finally,  those  with 
higher  perceived  attentional  control  performed  better  on 
a  concurrent  communication  task  in  the  more  difficult 
tasking  conditions.  Implications  for  military  personnel 
selection  were  discussed. 


1.  INTRODUCTION 
1.1  Background 

The  goal  of  this  research  was  to  examine  whether 
gunners  in  a  U.S.  Army  Future  Combat  System  (FCS) 
manned  vehicle  were  able  to  effectively  maintain  local 
security  (i.e.,  perform  their  gunner’s  tasks)  while 
managing  their  unmanned  assets  and  if/how  individual 
difference  factors  such  as  attentional  control  and  spatial 
ability  had  any  impact  on  the  performance.  Mitchell 
(2005)  examined  workload  for  the  Mounted  Combat 
System  (MCS)  crew  members  using  a  task-network 
modeling  tool.  Improved  Performance  Research 
Integration  Tool  (IMPRINT).  According  to  Mitchell,  the 
gunner  is  the  most  viable  option  for  controlling  robotic 
assets  compared  to  the  other  two  positions  (i.e., 


commander  and  driver).  She  found  that  the  gunner  had 
the  fewest  instances  of  overload  and  could  assume 
control  of  the  robotic  tasks.  However,  she  also 
discovered  that  there  were  instances  in  the  model  when 
the  gunner  dropped  his  primary  tasks  of  detecting  and 
engaging  targets  to  perform  robotic  control  tasks,  which 
could  be  catastrophic  for  the  team  and  mission  during  a 
real  operation. 

Past  research  in  dual  task  performance  suggests  that 
operators  may  encounter  difficulties  when  both  tasks 
involve  focal  vision  (Horrey  &  Wickens,  2004).  Horrey 
and  Wickens  (2004)  demonstrated  that  participants  could 
not  effectively  detect  road  hazards  while  operating  in¬ 
vehicle-devices.  Additionally,  research  on  visual 
performance  demonstrated  that  as  the  size  of  the  search 
set  increased,  performance  degraded  in  terms  of  either 
speed  or  accuracy  or  both  (Scanlan,  1977).  Murray 
(1994)  showed  that  as  the  number  of  monitored  displays 
increased,  operators’  reaction  time  for  their  target  search 
tasks  also  increased  linearly.  In  fact,  reaction  time  almost 
doubled  when  the  number  of  displays  increased  from  1  to 
2  and  from  2  to  3  (a  slope  of  1 .94  was  obtained). 

According  to  Wickens,  Dixon,  and  Chang  (2003), 
visual  angle  separation  larger  than  about  6.4  ~  7.5 
degrees  may  degrade  event  monitoring  response  time.  In 
the  case  of  concurrent  performance  of  gunner’s  and 
robotic  operator’s  tasks,  it  was  expected  that 
performance  would  be  worse  than  when  the  operator 
only  had  to  perform  one  task  since  concurrent  tasks 
involved  more  displays  to  visually  scan.  It  was  expected 
that  the  gunner’s  task  performance  would  further  degrade 
when  the  robotic  tasks  became  more  challenging.  For 
example,  when  robots  needed  teleoperation  and/or  when 
the  operator  needed  to  use  the  user  interface  to  perform 
some  tasks  (e.g.,  putting  targets  on  the  map,  labeling  the 
targets,  sending  spot-reports,  etc.).  Moreover,  research 
has  shown  that  increased  mental  workload  could  reduce 
the  size  of  operator’s  visual  field  (Rantanen  &  Goldberg, 
1999).  It  was  expected  that  the  reduced  visual  field 
would  have  a  significant  impact  on  the  operator’s 
gunnery  task  performance  (i.e.,  target  detection  in  his 
immediate  environment). 
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1.2  Current  Study 

This  current  research  tried  to  verify  the  modeling 
project’s  analytical  results  (Mitchell,  2005)  and 
examined  whether  the  gunner  could  effectively  detect 
targets  in  his  or  her  immediate  environment  while 
operating  robotic  assets  in  a  remote  environment.  In  this 
study,  we  simulated  a  generic  mounted  crewstation 
environment  and  conducted  an  experiment  to  examine 
the  workload  and  performance  of  the  combined  position 
of  gunner  and  robotic  operator.  The  experimental 
conditions  included  a  Gunner  Baseline  condition  and  3 
concurrent  task  conditions  where  participants 
simultaneously  performed  gunnery  tasks  and  one  of  the 
following  tasks:  monitor  an  unmanned  ground  vehicle 
(UGV)  via  the  video  feed  (Monitor  condition),  manage  a 
semi-autonomous  UGV  (UGV  condition),  and 
teleoperate  a  UGV  (Teleop  condition). 


small  UGV  had  a  higher  level  of  automation.  Luck  et  al. 
suggested  that  the  attention  on  (manual)  robotic  control 
might  have  distracted  the  operators  from  focusing  on  the 
vehicle’s  location,  which  was  the  study’s  measure  of 
situational  awareness.  Dixon,  Wickens,  and  Chang 
(2003)  also  reported  that  pilots  found  more  targets  when 
their  unmanned  aerial  vehicle(s)  were  autonomous  than 
when  they  were  teleoperated. 

Finally,  the  relationship  between  participants’  spatial 
ability  (SpA)  and  their  task  performance  was  examined. 
According  to  Chen  et  al.  (in  press),  those  with  higher 
SpA  performed  target  detection  tasks  using  robotic  assets 
more  effectively  than  those  with  lower  SpA.  In  the 
current  study,  two  different  types  of  spatial  tests  were 
employed.  It  was  expected  that  those  with  higher  SpA 
test  scores  would  perform  their  robotic  tasks  better. 


Participants  also  performed  a  tertiary 
communication  task  concurrently,  which  simulated 
gunner’s  communication  with  fellow  crew  members  in 
the  vehicle.  Richard  et  al.  (2002)  found  that  participants’ 
change  detection  was  negatively  affected  by  a  concurrent 
auditory  task.  More  specifically,  participants’  reaction 
times  were  slower  and  visual  scanning  was  less  effective. 
In  the  current  study,  we  expected  the  concurrent 
communication  to  have  a  similar  negative  effect  on 
participants’  target  detection  performance.  Although  we 
did  not  manipulate  the  communication  task  as  a  variable, 
we  tried  to  examine  if  participants  with  higher  attentional 
control  could  perform  their  tasks  more  effectively  than 
those  with  lower  attentional  control  in  our  simulated 
multi-tasking  environment.  Schumacher  et  al.  (2001) 
demonstrated  that  some  participants  were  more  effective 
in  concurrently  performing  a  visual  task  and  an  auditory 
task  but  did  not  examine  what  individual  difference 
factor(s)  contributed  to  that  time  sharing  effectiveness. 

In  the  current  study,  it  was  expected  that 
performance  would  be  worse  in  the  concurrent  task 
conditions  because  of  the  divided  visual  attention,  and 
that  the  gunner’s  task  performance  would  further  degrade 
when  the  robotic  tasks  became  more  challenging  (i.e., 
when  more  than  mere  monitoring  was  needed). 
Participants’  robotic  task  performance  was  expected  to 
differ  depending  on  the  type  of  asset  available  and  the 
type  of  task  they  were  asked  to  perform.  Chen,  Durlach, 
Sloan,  and  Bowens  (in  press)  demonstrated  that 
participants’  target  detection  was  significantly  lower 
when  they  had  to  teleoperate  the  unmanned  ground 
vehicle  (UGV)  as  compared  to  when  the  UGV  was  semi- 
autonomous.  Chen  et  al.  suggested  that  maybe 
participants’  teleoperation  (i.e.,  driving  the  robot) 
negatively  affected  their  target  detection  performance. 
Luck,  Allender,  and  Russell  (2006)  reported  that  robotic 
operators’  situational  awareness  was  better  when  the 


2.  METHOD 


2.1  Participants 

A  total  of  20  students  (3  females  and  17  males)  were 
recruited  from  the  University  of  Central  Florida  and 
participated  in  the  study.  The  ages  of  the  participants 
ranged  from  18  to  45  ( M  =  20.8,  SO  =  3.2).  Participants 
were  compensated  $40  and  class  credit  for  their 
participation  in  the  experiment. 

2.2  Apparatus 
2.2.1  Simulators 

The  experiment  was  conducted  using  the  Tactical 
Control  Unit  (TCU)  developed  under  Army  Research 
Laboratory’s  Robotics  Collaborative  Technology 
Alliance  (RCTA)  for  the  robotic  control  tasks.  The 
gunnery  component  was  implemented  using  an 
additional  screen  and  controls  to  simulate  the  out-the- 
window  view  and  both  line-of-sight  (LOS)  and  beyond- 
line-of-sight  (BLOS)  fire  capabilities  (see  Fig.  1). 


The  RCTA  TCU  is  a  one-person  crew  station  from 
which  the  operator  can  control  several  simulated  robotic 
assets,  which  can  either  perform  their  tasks  semi- 
autonomously  or  be  teleoperated.  The  operator  switched 
operation  modes  and  display  modes  through  the  use  of  a 
19”  touch-screen  display.  A  joystick  was  used  to 
manipulate  the  direction  in  which  the  unmanned  vehicles 
moved  when  in  Teleop  mode.  The  UGV  simulated  in  our 
study  is  the  experimental  unmanned  vehicle  (XUV) 
developed  by  the  Army  Research  Laboratory.  The 
simulation  program  used  in  this  study  was  rSAF,  which 
is  a  version  of  OneSAF  for  robotics  simulation. 

The  gunnery  component  consisted  of  a  monitor  and 
a  joystick.  The  interface  consists  of  a  15”  KOGi  flat 
panel  monitor  and  a  joystick.  Participants  used  the 
joystick  to  rotate  the  sensors  360  degrees,  zoom  in  and 
out,  switch  between  firing  modes,  and  engage  targets. 
For  engaging  BLOS  targets,  the  participants  would  need 
to  receive  authorization  from  the  vehicle  commander 
(i.e.,  the  experimenter)  and  then  line  up  his  or  her  aim 
with  the  direction  of  the  target  (the  line  would  turn  red 
when  it  was  aiming  at  the  target)  and  then  fire. 

Cognitive  tests  were  administered  concurrently  with 
the  experimental  sessions.  The  questions  included  simple 
military-related  reasoning  tests  and  simple  memory  tests. 
The  inclusion  of  these  cognitive  tasks  was  for  simulating 
an  environment  where  the  gunner  was  communicating 
with  fellow  crew  members  in  the  vehicle.  For  the 
reasoning  tests,  there  were  questions  such  as  “if  the 
enemy  is  to  our  left,  and  our  UGV  is  to  our  right,  what 
direction  is  the  enemy  to  the  UGV?”  For  the  memory 
tests,  the  participants  were  asked  to  repeat  some  short 
statements  or  keep  track  of  three  radio  call  signs  (e.g., 
Bravo  83)  and  they  had  to  report  to  the  experimenter 
whether  the  call  signs  they  heard  were  one  of  those  they 
were  keeping  track  of.  Test  questions  were  delivered  by 
a  synthetic  speech  program,  DECTalk®.  The  questions 
were  pre-recorded  by  a  male  speaker  and  presented  at  the 
rate  of  one  question  approximately  every  33  seconds. 

2.2.2  Questionnaires  and  Tests 

A  demographics  questionnaire  was  administered  at 
the  beginning  of  the  training  session.  A  questionnaire  on 
Attentional  Control  (Derryberry  &  Reed,  2002)  was  used 
to  evaluate  participants’  perceived  attentional  control. 
The  Attentional  Control  survey  consists  of  2 1  items  and 
measures  attention  focus  and  shifting.  The  Cube 
Comparison  Test  (Educational  Testing  Service,  2005) 
and  the  Spatial  Orientation  Test  were  used  to  assess 
participants’  spatial  ability.  The  Cube  Comparison  Test 
required  participants  to  compare,  in  3 -minutes,  21  pairs 
of  6-sided  cubes  and  determine  if  the  rotated  cubes  were 
the  same  or  different.  The  Spatial  Orientation  Test, 
constructed  by  Dr.  Paula  Durlach  of  the  U.S.  Army 


Research  Institute,  is  modeled  after  the  cardinal  direction 
test  developed  by  Gugerty  and  his  colleagues  (Gugerty  & 
Brooks,  2004)  and  is  a  computerized  test  consisting  of  a 
brief  training  segment  and  32  test  questions.  Both 
accuracy  and  response  time  were  automatically  captured 
by  the  program.  Participants’  perceived  workload  was 
evaluated  using  the  NASA-TLX  questionnaire  (Flart  & 
Staveland,  1988).  The  NASA-TLX  is  a  self-reported 
questionnaire  of  perceived  demands  in  nine  areas: 
mental,  physical,  temporal,  effort  (mental  and  physical), 
frustration,  performance,  visual,  cognitive,  and 
psychomotor.  Participants  were  asked  to  evaluate  their 
perceived  workload  level  in  these  areas  on  10-point 
scales. 

The  Simulator  Sickness  Questionnaire  (SSQ)  was 
used  to  evaluate  participants’  simulator  sickness 
symptoms  (Kennedy,  Lane,  Berbaum,  &  Lilienthal, 
1993).  The  SSQ  consists  of  a  checklist  of  16  symptoms. 
Each  symptom  is  related  in  terms  of  degrees  of  severity 
(none,  slight,  moderate,  severe).  A  total  severity  score 
can  be  derived  by  a  weighted  scoring  procedure  and 
reflects  overall  discomfort  level. 

Finally,  a  usability  questionnaire  was  constructed, 
based  on  the  one  used  in  the  Unmanned  Combat 
Demonstration  (UCD)  study  (Kamsickas,  2003). 
Specifically,  the  questionnaire  included  the  following 
sections:  asset  summary,  Reconnaissance,  Surveillance, 
and  Target  Acquisition  (RSTA),  map  display, 
teleoperation,  reporting,  and  general  usability  of  the 
TCU.  Participants  indicated  their  level  of  agreement  with 
the  items  using  7-point  numerical  scales.  Participants 
were  also  given  an  opportunity  to  provide  comments  to 
support  or  clarify  their  numeric  responses.  The 
comments,  in  addition  to  the  numeric  responses, 
provided  the  researchers  with  further  insight  as  to  the 
participants’  opinions  about  the  crewstation. 

2.3  Procedure 

After  the  informed  consent  process,  participants 
filled  out  the  Attentional  Control  Survey  and  were 
administered  the  Cube  Comparison  Test  and  the  Spatial 
Orientation  Test.  After  these  tests,  participants  received 
training,  which  lasted  approximately  two  hours.  The 
experimental  session  took  place  on  a  different  day  but 
within  a  week  of  the  training  session.  Each  experimental 
session  lasted  approximately  15  minutes,  and  the  order  of 
experimental  conditions  was  counterbalanced  across 
participants.  For  the  Gunner  Baseline  condition,  the 
operator  performed  only  gunnery  tasks  (i.e.,  target 
detection  and  engagement).  In  the  remaining  conditions, 
the  operator  performed  gunnery  tasks  and  one  of  the 
following  robotic  tasks:  monitor  the  video  feed  and 
verbally  report  any  targets  detected  (Monitor  condition); 
monitor  the  video  feed,  examine  still  images  generated 


from  the  reconnaissance  scans,  which  were  enabled  by 
the  aided  target  recognition  (ATR)  capabilities,  and 
detect  targets  (UGV  conditions);  and  manually 
manipulate  the  UGV  along  a  predetermined  route  to 
detect  targets  (Teleop  conditions).  There  were  two- 
minute  breaks  between  experimental  sessions. 

While  the  participants  were  performing  gunnery 
and/or  robotic  control  tasks,  they  performed  the 
communication  tasks  (i.e.,  cognitive  test)  simultaneously. 
For  the  purpose  of  comparing  participants’  robotic 
control  task  performance  between  the  single-task  and 
concurrent-task  conditions,  we  added  a  UGV-Baseline 
condition  to  half  of  the  participants  and  Teleop-Baseline 
to  the  other  half  of  the  participants.  In  these  two 
conditions,  they  did  not  have  to  perform  the  gunnery 
tasks  and  only  had  to  simultaneously  perform  the  robotic 
control  tasks  and  the  communication  tasks. 

Participants  filled  out  the  NASA-TLX  after  the  four 
main  experimental  conditions  and  the  SSQ  at  the  end  of 
the  experimental  session. 

2.4  Experimental  Design  and  Measures 

The  overall  design  of  the  study  is  a  repeated- 
measures  design.  There  were  four  conditions: 

•  Gunnery  Baseline  (Gunner  Baseline) 

•  Concurrent  task  conditions: 

o  Monitor:  Gunnery  +  Monitoring  1 
Semi-autonomous  UGV  (Monitor) 
o  Gunnery  +  Monitoring  +  Control  of  1 
Semi-autonomous  UGV  (UGV) 
o  Gunnery  +  Monitoring  +  Teleoperating 
UGV  (Teleop) 

The  dependent  measures  include  mission 
performance  (i.e.,  number  of  targets  detected  in  the 
remote  environment  using  the  robotic  assets  and  number 
of  enemy  targets  detected  in  the  immediate 
environment),  communication  task  performance,  and 
perceived  workload. 

3.  RESULTS 

3.1  Target  Detection  Performance 

3.1.1  Gunnery  Tasks 

Correlations  between  participants’  gunnery  task 
performance  and  their  Attentional  Control  Survey,  Cube 
Comparison  Test,  and  Spatial  Orientation  Test  scores 
were  first  evaluated.  The  Spatial  Orientation  Test  scores 
were  found  to  be  the  most  accurate  predictor  of 
participants’  gunnery  performance,  with  Gunner 


Baseline,  Monitor,  and  Teleop  conditions  being 
significant  (p’s  <  .05).  Participants  were  then  designated 
as  high  SpA  or  low  SpA  based  on  their  Spatial 
Orientation  Test  scores.  A  mixed  Analysis  of  Variance 
(ANOVA)  was  performed,  with  the  Asset  condition 
(Gunner  Baseline,  Monitor,  UGV,  Teleop)  being  the 
within-subject  factor  and  SpA  (Spatial  Orientation  Test 
score)  as  the  between-subject  factor.  The  analysis 
revealed  that  Asset  condition  significantly  affected 
number  of  targets  detected,  F( 3,  16)  =  28.417,  p  <  .0001, 
with  Gunner  Baseline  being  the  highest  and  Teleop  being 
the  lowest.  Participants  with  higher  SpA  had 
significantly  higher  gunnery  task  performance  than  did 
those  with  lower  SpA,  F(\,  18)  =  8.76,/)  <  .005  (Fig.  2). 

Gunnery  Task  Performance 

Effects  of  Spatial  Ability  (Spatial  Orientation  T est  Score) 


Gunner  Monitor  UGV  Teleop 


Condition 

Fig.  2.  Gunnery  task  performance. 

3.1.2  Robotic  Tasks 

Correlations  between  participants’  target  detection 
performance  on  the  robotic  tasks  and  their  Attentional 
Control  survey,  Cube  Comparison  Test,  and  Spatial 
Orientation  Test  scores  were  first  evaluated.  No 
consistent  significant  correlations  were  observed. 

Two  repeated-measures  ANOVAs  were  performed 
to  compare  participants’  target  detection  performance  on 
the  robotic  tasks,  one  for  the  human  targets  and  the  other 
for  the  vehicle  targets.  The  first  analysis  showed  that 
there  were  significant  differences  among  the  Monitor, 
UGV,  and  Teleop  conditions  in  human  target  detection, 
F( 2,  18)  =  4.794,  p  <  .05,  with  UGV  being  the  lowest. 
Post-hoc  tests  (LSD)  showed  that  differences  between 
Monitor  and  UGV  and  between  Teleop  and  UGV  were 
both  significant.  The  second  analysis  examined  the 
difference  between  the  UGV  and  the  Teleop  conditions 
in  vehicle  target  detection  and  the  difference  was  not 
significant. 


To  further  examine  why  the  Teleop  condition 
produced  better  target  detection  rates  than  the  UGV 
condition,  which  conflicted  with  the  findings  in  Chen  et 
al.  (in  press),  we  compared  the  number  of  targets 
detected  along  the  route  and  targets  detected  within 
RSTA  areas  for  the  UGV  and  Teleop  conditions.  A  2  x  2 
repeated  measures  ANOVA  was  performed  with  Asset 
condition  (UGV  vs.  Teleop)  and  Target  Location  (Route 
vs.  RSTA)  as  the  factors.  The  analysis  revealed  that  both 
effects  were  significant:  Asset,  F(  1,  38)  =  5.75,  p  =  .019; 
Location  F{\,  38)  =  18.01,  p  <  .0001.  Post-hoc  tests 
showed  that  the  largest  difference  in  targets  detected  was 
along  the  route,  with  UGV  having  a  35%  and  Teleop 
51%  target  detection  rate. 

Half  of  the  participants  also  completed  the  UGV- 
Baseline  condition  and  the  other  half  Teleop-Baseline 
condition  so  we  could  compare  if  participants  target 
detection  performance  degraded  when  they  had  to 
perform  the  gunnery  task  concurrently.  Results  showed 
that  when  participants  only  had  to  operate  the  UGV,  their 
overall  target  detection  rate  (including  both  human  and 
vehicle  targets)  was  80%;  when  they  had  to  concurrently 
operate  the  UGV  while  performing  the  gunner’s  tasks 
(i.e.,  UGV  condition),  their  target  detection  using  the 
UGV  dropped  to  67%  (difference  close  to  significance). 

3.2  Communication  Task  Performance 

The  differences  in  participants’  communication  task 
performance  among  the  four  conditions  were  significant, 
F( 3,  16)  =  6.574 ,p  <  .005,  with  the  Gunner  Baseline  and 
Monitor  conditions  being  higher  than  the  other  two 
conditions.  Participants’  perceived  attentional  control 
scores  were  found  to  be  positively  correlated  with  the 
communication  performance  in  the  Teleop  conditions,  r 
=  .476  (p  =  .023).  For  the  UGV  condition,  the  correlation 
was  close  to  significance,  r=  .385  {p  =  .057). 

3.3  Perceived  Workload 

Participants’  self-assessment  of  workload  was 
significantly  affected  by  Asset  condition,  F( 3,  17)  = 
65.102,  p  <  .0001,  with  Teleop  ( M  =  43.03)  being  the 
highest  and  Gunner  Baseline  ( M  =  22.35)  being  the 
lowest  (Fig.  3).  The  only  individual  difference  factor  that 
was  significant  was  participants’  Cube  Comparison  Test 
score,  FIT,  18)  =  6.995,  p  <  .05.  Those  with  higher  test 
scores  had  higher  perceived  workload  than  did  those 
with  lower  scores. 

3.4  Simulator  Sickness 

Participants’  Simulator  Sickness  Questionnaire 
(SSQ)  scores  were  calculated  based  on  the  formulae  in 
Kennedy  et  al.  (1993).  The  average  total  severity  score 
was  29.36  ( SD  =  24.06).  Further  examination  of  the  sub- 


Fig.  3.  Perceived  workload. 

scale  data  indicated  that  the  oculomotor  aspect 
significantly  contributed  to  the  elevated  total  severity 
score.  Correlations  between  SSQ  scores  and  perceived 
attentional  control  were  consistent  and  mostly 
significant,  r  =  -.612,  r  =  -.425,  r  =  -.432  {p  =  .003,  p  = 
.035,  p  =  .032)  for  nausea,  oculomotor,  and  total  severity 
score,  respectively.  Correlations  between  SSQ  and 
Spatial  Orientation  Test  scores  were  also  all  negative  but 
not  significant  (all  p’s  >  .05). 

3.5  Usability  Questionnaire 

Generally,  the  TCU  was  perceived  to  be  user 
friendly.  However,  a  number  of  participants  did  suggest 
that  it  required  too  many  steps  to  complete  some  simple 
tasks  and  they  needed  to  go  to  different  screens  to 
complete  those  steps  (e.g.,  putting  targets  on  the  map, 
labeling  the  targets,  and  sending  spot  reports).  While 
they  were  completing  those  steps,  they  could  not 
effectively  monitor  the  gunnery  station,  if  these  steps  can 
be  more  consolidated  and  centralized,  this  part  of  the 
robotic  task  would  not  require  as  much  visual  attention 
as  it  currently  does. 

4.  DISCUSSION 

In  this  study,  we  simulated  a  generic  mounted 
crewstation  environment  and  performed  an  experiment  to 
examine  the  workload  and  performance  of  the  combined 
position  of  gunner  and  robotic  operator.  Results  showed 
that  gunner’s  target  detection  performance  degraded 
significantly  when  he  or  she  had  to  concurrently  monitor, 
manage,  or  teleoperate  a  UGV.  The  gunner’s 
performance  in  the  three  concurrent-task  conditions  was 
all  significantly  different  from  one  another,  with  the 
Monitor  condition  being  the  highest  and  Teleop 
condition  being  the  lowest.  Participants’  SpA  as 
measured  by  their  Spatial  Orientation  Test  scores  was 
found  to  be  an  accurate  predictor  of  their  gunnery 


performance.  These  results  suggest  that,  if  it  is  necessary 
for  the  gunner  to  concurrently  access  information  from 
the  robotic  assets,  the  robotic  tasks  should  be  limited  to 
activities  such  as  monitoring.  If  excessive  manipulation 
of  the  Warfighter-machine  interface  is  required  as  in  the 
UGV  and  Teleop  conditions,  their  gunnery  performance 
will  be  significantly  affected.  When  selecting  personnel 
for  these  tasks,  it  might  be  beneficial  to  take  into  account 
their  SpA.  Thomas  and  Wickens  (2004)  showed  that 
there  were  individual  differences  in  scanning 
effectiveness  and  its  associated  target  detection 
performance.  However,  Thomas  and  Wickens  (2004)  did 
not  examine  the  characteristics  of  those  participants  who 
had  more  effective  scanning  strategies.  Results  of  the 
current  study  suggest  that  spatial  tests  like  the  Spatial 
Orientation  Test  might  be  useful  in  examining  individual 
differences  in  scanning  behavior  and  target  detection 
performance. 

Further  research  should  also  examine  the  difference 
between  the  Spatial  Orientation  Test  and  the  Cube 
Comparison  Test.  Based  on  the  results  of  the  current 
study,  it  appears  that  these  two  tests  tapped  different 
aspects  of  SpA.  Some  cognitive  modeling  work  done  by 
the  Army  Research  Laboratory,  Human  Research  & 
Engineering  Directorate  researchers  suggested  that  the 
Cube  Comparison  Test  may  reflect  ability  more 
associated  with  feature  comparison  than  with  spatial 
rotation  (Kelley,  Wiley,  &  Lee,  2000).  Verbal  protocol 
obtained  from  research  participants  in  Kelley  et  al. 
indicated  that  participants  used  a  variety  of  problem¬ 
solving  strategies  for  their  spatial  rotation  test  instead  of 
mentally  rotating  the  images. 

For  the  robotic  tasks,  there  were  significant 
differences  among  the  Monitor,  UGV,  and  Teleop 
conditions  in  human  target  detection  performance,  with 
the  UGV  being  the  lowest  (only  53%  were  detected). 
The  inferior  performance  associated  with  the  semi- 
autonomous  UGV  seemed  to  reflect  participant’s  over¬ 
reliance  on  the  aided  target  recognition  (ATR) 
capabilities  and  failure  to  detect  more  targets  along  the 
route  that  were  not  picked  up  by  the  ATR.  In  contrast,  in 
Chen  et  al.  (in  press),  participants  had  the  lowest  target 
detection  using  the  Teleop.  However,  in  Chen  et  al.’s 
UGV  condition,  the  ATR  capabilities  were  not  available. 
Results  of  the  current  study  are  consistent  with 
automation  research  that  operators  may  develop  over¬ 
reliance  on  the  automatic  system  and  this  complacency 
may  negatively  affect  their  task  performance 
(Parasuraman,  Molloy,  &  Singh,  1993).  Thomas  and 
Wickens  (2000)  showed  that,  when  participants  had 
access  to  information  gathered  from  automatically 
panning  cameras,  they  tended  to  prematurely  close  the 
automatic  panning  feature  prior  to  finishing  examining 
the  entire  environment.  Participants  manually  panning 
the  cameras,  on  the  other  hand,  had  significantly  higher 


target  detection  performance,  which  indicated  more 
adequate  panning.  It  is  worth  noting  that  these  findings 
along  with  the  results  of  the  current  study  do  not 
necessarily  suggest  that  manual  manipulation  of  sensor 
devices  be  used  instead  of  ATR  or  auto-panning  devices. 
However,  the  issue  of  over-reliance  on  these  automatic 
capabilities  needs  to  be  taken  into  account  when 
designing  the  Warfighter-machine  interface  where  these 
features  are  one  of  the  components. 

When  participants  only  had  to  operate  the  UGV  (i.e., 
UGV  baseline),  their  overall  target  detection  rate  was 
80%;  when  they  had  to  concurrently  operate  the  UGV 
while  performing  the  gunner’s  tasks  (i.e.,  UGV 
condition),  their  target  detection  using  the  UGV  dropped 
to  67%.  It  also  appears  that  the  performance  difference 
between  UGV  and  Teleop  widened  in  the  concurrent  task 
conditions  compared  with  the  single-task  (baseline) 
conditions.  In  other  words,  the  UGV -Baseline  and 
Teleop-Baseline  were  at  similar  levels  but  UGV- 
Concurrent  was  significantly  lower  than  Teleop- 
Concurrent.  These  results  suggest  that  as  operator’s  tasks 
become  more  challenging  (i.e.,  concurrent  conditions), 
he  or  she  may  rely  more  on  the  ATR  capabilities  to 
relieve  the  workload  if  they  are  available.  However,  the 
over-reliance  on  the  ATR  capabilities  may  result  in 
overall  performance  degradation  as  discussed  in  the 
previous  paragraph. 

None  of  the  individual  difference  factors  were  found 
to  significantly  correlate  with  participants’  robotic  task 
performance.  The  lack  of  correlation  between  SpA  and 
robotic  task  performance  was  unexpected.  Chen  et  al.  (in 
press)  showed  that  those  with  higher  SpA  (as  measured 
by  the  Cube  Comparison  Test)  had  significantly  higher 
performance  in  their  target  detection  task  using  the  UGV. 
Further  research  is  needed  to  examine  the  relationship 
between  SpA  and  robotic  task  performance. 

Participants’  communication  task  performance 
degraded  when  their  robotic  tasks  became  more 
challenging  (i.e.,  UGV  and  Teleop  conditions).  It  is 
interesting  to  note  that  participants  appeared  to  be  able  to 
perform  their  communication  task  at  similar  levels  in  the 
Gunner  Baseline  and  Monitor  conditions.  This  suggests 
that,  in  the  Monitor  condition,  participants  had  sufficient 
cognitive  resources  left  to  perform  the  communication 
tasks.  Participants  with  higher  perceived  attentional 
control  performed  better  on  the  concurrent 
communication  task,  although  they  performed  at  a 
similar  level  on  their  gunnery  and  robotic  control  tasks  as 
those  with  lower  perceived  attentional  control.  These 
results  suggest  that  participants  devoted  most  of  their 
attention  resources  to  the  gunnery  and  robotic  tasks,  and 
only  those  with  higher  attention  allocation  skills  could 
more  successfully  perform  the  tertiary  communication 
tasks.  Since  communication  will  be  a  critical  part  in  the 


task  environment,  these  results  may  have  important 
implications  for  personnel  selection  for  the  Army’s 
future  forces. 

Participants’  perceived  workload  increased  almost 
linearly  in  order  from  the  Gunner  Baseline,  Monitor, 
UGV  and  to  the  Teleop  condition,  and  the  differences 
among  the  four  conditions  were  all  statistically 
significant.  These  results  are  consistent  with  Schipani 
(2003),  which  evaluated  robotic  operator  workload  in  a 
field  setting.  Although  many  of  the  ground  robotic  assets 
in  the  Army’s  FCS  program  will  be  semi-autonomous,  it 
is  very  likely  that  teleoperation  will  be  required  at  times 
when  the  robotic  assets  encounter  problems.  The  higher 
workload  associated  with  teleoperation  needs  to  be  taken 
into  account  when  designing  the  Warfighter-machine 
interfaces  for  the  FCS.  Additionally,  it  appears  that 
participants  with  higher  SpA,  although  performing  better 
on  the  tasks,  did  not  perceive  the  tasks  as  less 
demanding.  In  fact,  the  correlations  between 
participants’  perceived  workload  and  their  Cube 
Comparison  Test  scores  or  Spatial  Orientation  Test 
reaction  times  indicated  that  those  with  higher  SpA  (at 
least  according  to  these  two  measures)  actually  perceived 
the  tasks  as  more  demanding.  The  correlations  between 
participants’  perceived  attentional  control  and  their 
workload  were  as  expected.  Those  with  higher 
attentional  control  thought  the  tasks  as  less  demanding.  It 
is  worth  noting  that  only  in  the  most  challenging 
condition  (i.e.,  Teleop)  did  the  correlation  reach 
significance.  In  other  words,  as  the  tasks  became  harder, 
the  differences  in  the  levels  of  perceived  workload 
between  those  with  higher  and  lower  perceived 
attentional  control  appeared  to  widen.  The  relationships 
between  workload  and  perceived  attentional  control  and 
SpA  need  to  be  further  investigated. 

Participants  seemed  to  experience  somewhat 
significant  levels  of  simulator  sickness,  especially  in  the 
oculomotor  area.  The  high  demand  of  visual  attention, 
particularly  in  the  concurrent  task  conditions,  may  have 
contributed  to  the  elevated  levels  of  simulator  sickness 
the  participants  experienced.  It  is  worth  noting  that  our 
entire  experimental  session  only  lasted  about  one  and  a 
half  hours.  Any  duration  longer  than  this  may  induce 
even  more  severe  sickness.  Participants  with  lower 
perceived  attentional  control  had  significantly  higher 
simulator  sickness  than  did  those  with  higher  attentional 
control.  It  was  reported  that,  in  the  virtual  environment, 
those  who  need  to  concentrate  more  tended  to  experience 
higher  levels  of  simulator  sickness  (Regan,  as  cited  in 
Kolasinski,  1995).  The  findings  of  the  Regan  study  and 
the  current  study  seem  to  suggest  that  those  who  are 
better  at  allocating  their  attentional  resources  may 
experience  lower  simulator  sickness.  Focused  attention 
has  been  found  to  positively  correlate  with  telepresence 
(Novak,  Floffman,  &  Yung,  2000),  which  is  negatively 


correlated  with  simulator  sickness  (Witmer  &  Singer, 
1998).  Alternatively,  those  with  lower  perceived 
attentional  control  might  pay  more  attention  to  (and  be 
distracted  by)  their  own  bodily  reaction.  This  increased 
awareness  may  have  contributed  to  the  elevated  levels  of 
simulator  sickness.  It  is  also  possible  that  they  were  not 
experiencing  more  sickness  but  were  simply  more  aware 
of  the  symptoms.  More  research  in  this  area  is  needed. 

CONCLUSIONS 

The  findings  of  the  current  study  suggest  that,  in  an 
FCS  multi-tasking  environment,  if  it  is  necessary  for  the 
gunner  to  concurrently  access  information  from  the 
robotic  assets,  the  robotic  tasks  should  be  limited  to 
activities  such  as  monitoring.  If  excessive  manipulation 
of  the  Warfighter-machine  interface  is  necessary,  as  in 
the  UGV  and  Teleop  conditions  of  this  study,  their 
gunnery  performance  and  communications  with  fellow 
crew  members  may  be  significantly  affected.  When 
selecting  personnel  for  these  dual  positions,  it  might  be 
beneficial  to  consider  operator’s  spatial  ability  and 
perceived  attentional  control.  Follow  on  studies  will 
examine  the  utility  of  ATR-enabled  alert  for  improving 
gunner  dual  task  performance  and  will  vary  the  type  of 
alert  (tactile  vs.  visual)  and  the  reliability  of  the  ATR 
(miss-prone  vs.  false  alarm-prone). 

The  results  of  this  experiment  were  consistent  with 
the  predictions  generated  by  the  modeling  analysis  in 
Mitchell  (2005).  More  specifically,  Mitchell  predicted 
that  the  gunner  would  become  overloaded  if  s/he  has  to 
perform  the  gunnery  duties  while  concurrently  operating 
unmanned  assets,  especially  when  teleoperation  is 
required.  Moreover,  the  gunner,  when  overloaded,  may 
reprioritize  the  tasks  and  give  higher  priority  to  dealing 
with  the  robot  and  s/he  will  likely  scan  less  and, 
therefore,  jeopardize  the  security  of  the  platform.  Our 
data  confirmed  these  predictions.  In  addition,  we 
examined  how  various  individual  differences  factors  may 
affect  the  operator’s  performance.  Our  data  on  individual 
differences  can  be  used  in  future  MCS  modeling  efforts 
as  input  data  to  modeling  tasks  and,  therefore,  enhance 
future  model  analyses  (Mitchell  &  Chen,  2006).  The 
findings  of  our  study  have  also  been  forwarded  to  the 
MCS  design  group  so  the  data  can  be  incorporated  in 
their  manning  assessment  for  system  functional  review 
(Mitchell  &  Chen,  2006). 
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